fix(python, rust): expr parsing date/timestamp #2357
Conversation
ion-elgreco
requested review from
wjones127,
fvaleye,
roeap and
rtyler
as code owners
github-actions
bot
added
binding/python
ion-elgreco
force-pushed
the
fix/expr_parsing_date_timestamp
branch
2 times, most recently
from
3f8d32c
to
9c75bd0
Compare
ion-elgreco
force-pushed
the
fix/expr_parsing_date_timestamp
branch
from
9c75bd0
to
c206647
Compare
…ion-elgreco/delta-rs into fix/expr_parsing_date_timestamp
There was a problem hiding this comment.
The existing unit tests should be extended to ensure we can round trip these new types. https://github.com/delta-io/delta-rs/pull/2357/files#diff-8e1a60f799cd1ad29923ee75871099a19c15ddfc00bef4617d82bdc02cbe3198R523-R526
|
@Blajda that datafusion sql parser is for some reason parsing a date/timestamp as a substraction : S I also tried wrapping it in single quotes but then it gets parsed as a string. So I am not sure what format datafusion expects datetimes to be left: BinaryExpr(BinaryExpr { left: Column(Column { relation: None, name: "_timestamp_ntz" }), op: Gt, right: Literal(TimestampMicrosecond(1262304000000000, None)) })
right: BinaryExpr(BinaryExpr { left: Column(Column { relation: None, name: "_timestamp_ntz" }), op: Gt, right: BinaryExpr(BinaryExpr { left: BinaryExpr(BinaryExpr { left: Literal(Int64(2010)), op: Minus, right: Literal(Int64(1)) }), op: Minus, right: Literal(Int64(1)) }) }) |
|
Yeah you can't dump it directly. The query dialect is postgress so you if you are using a string you can try the cast notation i.e '::date'. Alternative approach use the date functions and convert the scalars to their correct representations. |
|
@Blajda yeah but then Im not really testing round tripping anymore. Im assuming the string format needs to be in the same format as the serialization format for the conflict resolution across other engine writers |
|
There's no specification on the query format for conflict resolution with other engines. At least for our engine any queries we convert to a string we should be able to read it back without error |
Description
We weren't parsing all scalar values yet, parses date32/64 and timestampmicros now as well.
Related Issue(s)